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DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.1 14. Applicant's submission filed on 01/26/2009 has been entered. 

Response to Amendment 

2. The amendments, filed 01/26/09, have been entered and made of record. Claims 1-3, 5-7, 
9-20, 22-28 and 30-33 are pending and considered on the merits. 

Claim Rejections - 35 USC § 112 

3. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

4. Claims 1-3, 5-7, 9-20, 22-28 and 30-33 are rejected under 35 U.S.C. 1 12, second 
paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject 
matter which applicant regards as the invention. 

Regarding claim 1, on line 10 the phrase "a first specifying unit configured to specify a 
position and an orientation " (emphasis added) renders the claim indefinite because it is unclear 
which element's position and orientation the first specifying unit specifies. See MPEP 
§ 2173.05(d). 
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Regarding claim 5, on line 3 the phrase "registering unit" renders the claim indefinite 
because it is unclear which registering unit, first or second, the claimed invention refers to. 

Claims 1-3, 6-7, 9-14 and 30-33 are rejected as they depend on the reject claim 1. 

Regarding claim 15, on line 6 the phrase "the specified position and orientation " (line 6) 
and " specifying a position and an orientation " (line 14) (emphasis added) renders the claim 
indefinite because it is unclear which element's position and orientation the specifying unit 
specifies. 

Regarding claim 16, on lines 1-2 the phrase "specifying a position and an orientation " 
(emphasis added) renders the claim indefinite because it is unclear which specifying unit the 
claim refers to. 

Regarding claim 17, on line 2 the phrase "said view" renders the claim indefinite because 
it is unclear which view, first or second, the claim refers to. 

Claims 18-20 and 22-27 are rejected as they depend on the reject claim 15. 

Regarding claim 28, on line 18 the phrase " specifying a position and an orientation " 
(emphasis added) renders the claim indefinite because it is unclear which element's position and 
orientation the specifying unit specifies. 

Claim Rejections - 35 USC §103 

5. The text of those sections of Title 35, U.S. Code not included in this action can be found 
in a prior Office action. 

6. Claims 1-3, 5-7, 9-20, 22-28 and 30 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Allard (US 6,535,793) in view of Ebersole et al. (US 2002/0010734). 
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Regarding Claim 1, Allard teaches a system for remote programming of an industrial 
robot (Fig. 1), the system comprising: a camera (Figs. 1-2, camera 140) for capturing an image, 
the camera being movably located on the robot (100) at a local site (col. 3:51 - col. 4:4; col. 
6:52-59; col. 7:45 - col. 8:15), 

a remote display device (heads-up display 310) located at a remote site, physically 
separated from the local site, for displaying a view received from the robot (col. 5:14 -65), 

a first specifying unit (Figs. 3-5, user interface 300) configured to specify a position and 
an orientation in the remote site in relation to a fixed remote coordinate system (col. 6:5-59; col. 
7:6 - col. 8:26), and wherein the display is adapted to display the image received from the robot 
in dependence on the position and orientation specified by the first specifying unit (Figs. 3-14; 
col. 9:3-47; col. 11:1 - col. 12:54), and 

a communication link configured to communicate information between the local site and 
the remote site and to communicate to the robot positions and orientations specified by the first 
specifying unit (col. 4:23-50). 

Allard fails to teach a first registering unit configured to generate graphics and register 
the graphics generated by the first registering unit on the image from the camera, to provide a 
composite augmented reality image, wherein the first registering unit is adapted to register the 
generated graphics to the augmented reality image in dependence on the position and orientation 
specified by the first specifying unit; a second specifying unit configured to specify a position 
and an orientation of the robot at the local site in relation to a local coordinate system, wherein a 
position and orientation of the robot is dependent on the position and orientation specified by the 
first specifying unit in the remote coordinate system; a second registering unit_configured to 
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generate graphics and register the generated graphics on an environment at the local site or an 
image of the environment of the local site, in dependence on the position and the orientation 
specified by the second specifying unit; and a local display device configured to display the 
environment at the local site and the graphics generated by the second registering unit projected 
on the environment. 

In the same field of endeavor, in figures 3-4 Ebersole teaches an augmented reality 
system comprising one or more local or remote AR station (6), wherein each local or remote AR 
station includes a first registering unit (Fig. 4, Video mixer 43 and converter 48) configured to 
generate graphics and register the graphics generated by the first registering unit on the image 
from the camera (34), to provide a composite augmented reality image fl| 0034, 0039), wherein 
the first registering unit is adapted to register the generated graphics to the augmented reality 
image in dependence on the position and orientation specified by the first specifying unit located 
in a local or remote non-AR station (5) (]f 0034-0036, 0040). Ebersole further teaches a second 
specifying unit (Fig. 3, tracking system 33 or Fig. 4, tracking station 47) configured to specify a 
position and an orientation of the robot at the local site in relation to a local coordinate system, 
wherein a position and orientation of the robot is dependent on the position and orientation 
specified by the first specifying unit in the remote coordinate system (]f 0058); a second 
registering unit (Fig. 4, local AR computer 31a) configured to generate graphics and register the 
generated graphics on an environment at the local site or an image of the environment of the 
local site, in dependence on the position and the orientation specified by the second specifying 
unit (]f 0034-0035); and a local display device (Fig. 3, 32 - AR display system or Fig. 4, head- 
mounted display 45) configured to display the environment at the local site and the graphics 
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generated by the second registering unit projected on the environment (]f 0028, 0035, 0038-0040, 
0058-0059). In light of the teaching from Ebersole, it would have been obvious to one skilled in 
the art at the time the invention was made to modify the method and system for remote control of 
mobile robot taught in Allard by having the first registering unit, the second specifying unit, the 
second registering unit and the local display device in order to provide users benefit up-to-date 
information received from remote computers or humans over a network 

Regarding Claim 2, Allard in view of Ebersole teaches the system according to claim 1 , 
wherein said first specifying unit comprises a tracking unit (Allard, Figs. 3-9, user interface 300 
& heads-up display 310) adapted to determine a position and orientation of a movable device 
located at the remote site (Allard, col. 5:13-65; col. 6:52 - col. 7:28; col. 7:45 - col. 8:15), the 
first registering unit adapted to register the generated graphics on the image in dependence of the 
position and orientation of the movable device (Ebersole, Fig. 4, head-mounted display 45, \ 
0034, 0036, 0039), and the camera is arranged such that its position and orientation are 
dependent on the position and orientation of the movable device (Ebersole teaches that the 
remote AR station can view any viewpoint which is created in AR using a camera or series of 
camera that are remotely controlled over the network by the remote viewer, ]|0040. In paragraph 
0057 Ebersole further teaches that both local and remote can be AR station which cameras can 
be obviously located at both local and remote stations). 

Regarding Claim 3, Allard in view of Ebersole teaches the system according to claim 2, 
wherein said movable device is the remote display device (Ebersole, Fib. 4, head-mounted 
display 45. It is inherent that the heads-mounted display 45 is movable). 
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Regarding Claim 5, Allard in view of Ebersole the system according to claim 1, further 
comprising a graphical generator (Allard, user interface 300 has overhead map 340, status bar 
350 and camera control 360) (Ebersole, Fig. 3, AR display system 32) configured to generate a 
graphical representation, wherein the registering unit is adapted to generate graphics based on the 
graphical representation (Allard, Figs. 3-14) (Ebersole, 0034-0035; In paragraph 0057 Ebersole 
further teaches that both local and remote can be AR station which cameras can be obviously 
located at both local and remote stations). 

Regarding Claim 6, Allard in view of Ebersole teaches the system according to claim 1, 
further comprising operator input means (Allard, Figs. 3, user interface 300) (Ebersole, Fig. 3, 
input devices 35 & 39) located at the remote site and configured to feed data related to the 
graphics to be displayed to the system, wherein the system is adapted to generate the graphics 
based on said data (Allard, col. 5:18-41; col. 6:5-51) (Ebersole, f)031, 0033). 

Regarding Claim 7, Allard in view of Ebersole the system according to claim 6, wherein 
said operator input means comprises a pointing device and a tracking unit configured to 
determine a position of the pointing device (Allard - joystick or direct drive controls 320 - col. 
5:20-29) (Ebersole - Fig. 3, input devices 35 & 39 and head-worn tracking device 47) and 
wherein the system is adapted to generate a graphical representation of a point pointed out by the 
pointing member based on the position of the pointing device (Allard, col. 5:18-41; col. 6:5-51) 
(Ebersole, ^0031,0033). 

Regarding Claim 9, Allard in view of Ebersole teaches the system according to claim 1, 
further comprising a second movable device located at the local site (Allard, heads-up display 
310), wherein the second specifying unit comprises a second tracking unit configured to 



Application/Control Number: 10/743,315 Page 8 

Art Unit: 2622 

determine the position and the orientation of the second movable device (Ebersole - Fig. 3, input 
devices 35 & 39) (Ebersole, H 0034-0035; In paragraph 0057 Ebersole further teaches that both 
local and remote can be AR station which cameras can be obviously located at both local and 
remote stations taught in Allard. Allard as modified by Ebersole would have taught a second 
head- worn tracking device that is used to determine the location & orientation of the camera). 

Regarding Claim 10, Allard in view of Ebersole teaches the system according to claim 9, 
wherein said second movable device is the local display device (Allard, heads-up display 310). 

Regarding Claim 11, Allard in view of Ebersole teaches the system according claim 9, 
further comprising a second camera for capturing an image, the camera being arranged in a fix 
relation to the second movable device (Ebersole, U 0034-0035; In paragraph 0057 Ebersole 
further teaches that both local and remote can be AR station which cameras can be obviously 
located at both local and remote stations taught in Allard), wherein the second registering unit is 
adapted to register the generated graphics generated by the second registering unit to the image 
from the second camera, to provide a composite augmented reality image, and wherein the local 
display device is adapted to display a view comprising the composite augmented reality image 
(Allard, col. 5:18-41; col. 6:5-51) (Ebersole, ^0031, 0033). 

Regarding Claim 12, Allard in view of Ebersole teaches the system according to claim 1 , 
wherein the remote display device (Allard, heads-up display 3 1 0) is adapted to display a view 
seen from a first visual angle that depends on the position and orientation received from the first 
specifying unit (Allard, Figs. 9-12; col. 8:41 - col. 9:47) and wherein the local display device 
(Ebersole, AR display system 32) is adapted to display the same view as the remote display 
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device seen from a second visual angle that depends on the position and orientation received 
from the second specifying unit (Ebersole, Tf 0036, 0038-0040, 0059). 

Regarding Claim 13, Allard in view of Ebersole teaches the system according to claim 1, 
wherein the communication link is configured to transfer voices between the remote and the 
local site (Ebersole, H 0060-0061). 

Regarding Claim 14, Allard in view of Ebersole teaches the system according to claim 1, 
wherein the communication link comprises a network (Allard, col. 4:23-50) (Ebersole, ](0032). 

Regarding Claim 30, Allard in view of Ebersole teaches the system according to claim 1, 
wherein the system is configured for remote programming of an industrial robot by controlling 
movements of the robot at the local site and teaching the robot one or more waypoints to carry 
out a task (Allard, Fig. 1, robot 100; col. 3:28— col. 4:4). 

Regarding Claims 15-20, 22-25 and 27-28, although the wording is different, the material 
is considered substantively equivalent to the material associated with claims 1-3, 5, 7, 9 and 11- 
12, respectively, as discussed above. 

Regarding Claim 26, Allard in view of Ebersole teaches the method according to claim 
25, further comprising generating a local graphical representation (Ebersole, Fig. 3, AR display 
system 32), generating a remote graphical representation (Allard, Figs. 3-14), transferring the 
local and remote graphical representations between the local and the remote site (Allard, col. 
4:23-50) (Ebersole, ](0032), generating the remote first graphics based on the local and the 
remote graphical representation (Allard, Figs. 3-14; col. 9:3-47; col. 11:1 - col. 12:54), and 
generating the second graphics based on the local and the remote graphical representation 
(Ebersole, If 0028, 0035, 0038-0040, 0058-0059) 
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7. Claims 3 1-32 are rejected under 35 U.S.C. 103(a) as being unpatentable over Allard in 
view of Ebersole as applied to claim 1 1 above, and further in view of Ellenby et al. (US 
5,815,411). 

Regarding Claim 31, Allard in view of Ebersole teaches the system according to claim 
11, however, Allard and Ebersole fail to teach a handheld display device comprising the display 
member and the camera. In the same field of endeavor, Ellenby teaches a vision system of 
image augmented reality wherein an image of a real scene is combined with information from a 
data base, wherein the system includes a handheld display device (Figs. 1-4 - binoculars being 
handheld) comprising a display member (13) and a camera (9). In light of the teaching from 
Ellenby, it would have been obvious to one of ordinary skill in the art at the time of the invention 
to allow the head mounted displays taught by Allard and Ebersole as viewed to be a handheld 
tracked display in order to offer the user a preferred method of viewing (Ellenby, col. 3:63, col. 
9:47-51). 

Regarding Claim 32, Ellenby teaches that the handheld display device is arranged so that 
the user seems to look directly through the display (col. 9:47-51; Claim 4). 

8. Claim 33 is rejected under 35 U.S.C. 103(a) as being unpatentable over Allard in view of 
Ebersole as applied to claim 1 above, and further in view of Nose et al. (US 5,079,491). 

Regarding Claim 33, Allard in view of Ebersole teaches the system according to claim 1, 
however, Allard and Ebersole fail to teach the robot comprises elements for a paint application. 
Nose teaches in figure 1 a robot control system including a robot (14), a robot controller (20) and 
a remote teaching box (Fig. 3 , box 1 8) for teaching the robot for paint application (col. 1 : 1 6-22; 
col. 5:36 - col. 6:3). In light of the teaching from Nose, it would have been obvious to one 
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skilled in the art at the time the invention was made to teach the robot taught in Allard for a paint 
application since Allard specifically teaches that the invention has broad applicability for the 
remote control of robotic devices (Allard, col. 1:11-25). 
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